Exploring the dynamics of the biocybernetic loop in physiological computing

Physiological computing is a highly multidisciplinary emerging field in which the spread of results across several application areas and disciplines creates a challenge of combining the lessons learned from various studies. The thesis comprises diverse publications that together create a privileged position for contributing to a common understanding of the roles and uses of physiological computing systems, generalizability of results across application areas, the theoretical grounding of the field (as with the various ways the psychophysiological states of the user can be modeled), and the emerging data analysis approaches from the domain of machine learning. The core of physiological computing systems has been built around the concept of biocybernetic loop, aimed at providing real-time adaptation to the cognitions, motivations, and emotions of the user. However, the traditional concept of the biocybernetic loop has been both self-regulatory and immediate; that is, the system adapts to the user immediately. The thesis presents an argument that this is too narrow a view of physiological computing, and it explores scenarios wherein the physiological signals are used not only to adapt to the user but to aid system developers in designing better systems, as well as to aid other users of the system. The thesis includes eight case studies designed to answer three research questions: 1) what are the various dynamics the biocybernetic loop can display, 2) how do the changes in loop dynamics affect the way the user is represented and modeled, and 3) how do the choices of loop dynamics and user representations affect the selection of machine learning methods and approaches? To answer these questions, an analytical model for physiological computing is presented that divides each of the physiological computing systems into five separate layers. The thesis presents three main findings corresponding to the three research questions: Firstly, the case studies show that physiological computing extends beyond the simple real-time self-regulatory loop. Secondly, the selected user representations seem to correlate with the type of loop dynamics. Finally, the case studies show that the machine learning approaches are implemented at the level of feature generation and are used when the loop diverges from the traditional real-time and self-regulatory dynamics into systems where the adaptation happens in the future.Perinteinen ihmisen ja tietokoneen vuorovaikutus on hyvin epäsymmetristä: tietokone voi esittää ihmiselle monimutkaista audiovisuaalista informaatiota kun taas ihmisen kommunikaatio koneen suuntaan on rajattu näppäimistöön ja hiireen. Samoin, vaikka ihmisellä on mahdollisuus saada informaatiota tietokoneen sisäisestä tilasta, kuten muistin ja prosessorin käyttöasteesta, ei tietokoneella ole vastaavaa mahdollisuutta tutkia ihmisen sisäisiä tiloja kuten tunteita. Mittaamalla reaaliajassa ihmisen fysiologisia signaaleja nämä molemmat ongelmat voidaan ratkaista: näppäimistön ja hiiren lisäksi tietokone saa suuren määrän informaatiota ihmisen kognitiivisista ja affektiivisista tiloista. Esimerkiksi mittaamalla ihmisen sykettä tai ihon sähkönjohtavuutta voi tietokone päätellä onko käyttäjä juuri nyt kiihtynyt tai rentoutunut. Tällaista fysiologisten signaalien reaaliaikaista hyödyntämistä ihmisen ja koneen vuorovaikutuksessa on tutkittu onnistuneesti monessa eri yhteyksissä: autonkuljettajien väsymystä voidaan mitata ja tarvittaessa varoittaa ajajaa, tietokonepelaajia mittaamalla on mahdollista säätää pelin vaikeustasoa sopivaksi ja älykello voi reagoida käyttäjän stressiin ehdottamalla rentoutumisharjoitusta. Näitä tapauksia yhdistää se, että käyttäjän fysiologisia signaaleja käytetään reaaliajassa sopeuttamaan järjestelmä käyttäjän itsensä tarpeisiin. Tällaista järjestelmän sopeuttamista reaaliajassa käyttäjän fysiologisten signaalien perusteella kutsutaan “biokyberneettiseksi silmukaksi” (biocybernetic loop). Biokyberneettisen silmukka on perinteisesti määritelty systeemin sopeuttamiseen yksittäisen käyttäjän sen hetkisen fysiologisen vasteen mukaan. Väitöskirjan tarkoitus on tutkia kuinka biokyberneettisen silmukan dynamiikkaa voidaan laajentaa sekä tilassa (voiko silmukka käsittää useita käyttäjiä) ja ajassa (voiko silmukan idea toimia myös ei-reaaliajassa). Erityisesti keskitytään tutkimaan kuinka muutokset silmukan dynamiikassa vaikuttavat silmukan toteutuksen yksityiskohtiin: kannattaako käyttäjää mallintaa eri tavoin ja ovatko tietyn tyyppiset silmukat soveltuvampia koneoppimiseen verrattuna ns. käsintehtyyn ratkaisuun. Väitöskirja sisältää kahdeksan käyttäjätutkimusta, jotka peilaavat biokyberneettisen silmukan käyttäytymistä erilaisissa konteksteissa. Tutkimukset osoittavat, että biokyberneettistä silmukkaa voidaan käyttää myös osana järjestelmän suunnittelua kun fysiologisten mittausten tulokset ohjataan järjestelmän kehittelijöille, ja järjestelmän muiden käyttäjien auttamiseen suosittelujärjestelmissä, joissa käyttäjän antamaa implisiittistä palautetta käytetään hyväksi suositeltaessa tuotteita toisille käyttäjille

Exploring Peripheral Physiology as a Predictor of Perceived Relevance in Information Retrieval

Peripheral physiological signals, as obtained using electrodermal activity and facial electromyography over the corrugator supercilii muscle, are explored as indicators of perceived relevance in information retrieval tasks. An experiment with 40 participants is reported, in which these physiological signals are recorded while participants perform information retrieval tasks. Appropriate feature engineering is defined, and the feature space is explored. The results indicate that features in the window of 4 to 6 seconds after the relevance judgment for electrodermal activity, and from 1 second before to 2 seconds after the relevance judgment for corrugator supercilii activity, are associated with the users’ perceived relevance of information items. A classifier verified the predictive power of the features and showed up to 14% improvement predicting relevance. Our research can help the design of intelligent user interfaces for information retrieval that can detect the user’s perceived relevance from physiological signals and complement or replace conventional relevance feedback

Effect of the Calibration in Water and the Build-up Cap on the Mg(Ar) Ionization Chamber Measurements

Non Peer reviewe

The influence of implicit and explicit biofeedback in first-person shooter games

To understand how implicit and explicit biofeedback work in games, we developed a first-person shooter (FPS) game to experiment with different biofeedback techniques. While this area has seen plenty of discussion, there is little rigorous experimentation addressing how biofeedback can enhance human–computer interaction. In our two-part study, (N=36) subjects first played eight different game stages with two implicit biofeedback conditions, with two simulation-based comparison and repetition rounds, then repeated the two biofeedback stages when given explicit information on the biofeedback. The biofeedback conditions were respiration and skin-conductance (EDA) adaptations. Adaptation targets were four balanced player avatar attributes. We collected data with psychophysiological measures (electromyography, respiration, and EDA), a game experience questionnaire, and game-play measures. According to our experiment, implicit biofeedback does not produce significant effects in player experience in an FPS game. In the explicit biofeedback conditions, players were more immersed and positively affected, and they were able to manipulate the game play with the biosignal interface. We recommend exploring the possibilities of using explicit biofeedback interaction in commercial games. Author Keywords Games, playing, affective computing, biosignals

Keep your opponents close : Social context affects EEG and fEMG linkage in a turn-based computer game

Peer reviewe

Predicting term-relevance from brain signals (Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval)

Term-Relevance Prediction from Brain Signals (TRPB) is proposed to automatically detect relevance of text information directly from brain signals. An experiment with forty participants was conducted to record neural activity of participants while providing relevance judgments to text stimuli for a given topic. High-precision scientific equipment was used to quantify neural activity across 32 electroencephalography (EEG) channels. A classifier based on a multi-view EEG feature representation showed improvement up to 17% in relevance prediction based on brain signals alone. Relevance was also associated with brain activity with significant changes in certain brain areas. Consequently, TRPB is based on changes identified in specific brain areas and does not require user-specific training or calibration. Hence, relevance predictions can be conducted for unseen content and unseen participants. As an application of TRPB we demonstrate a high-precision variant of the classifier that constructs sets of relevant terms for a given unknown topic of interest. Our research shows that detecting relevance from brain signals is possible and allows the acquisition of relevance judgments without a need to observe any other user interaction. This suggests that TRPB could be used in combination or as an alternative for conventional implicit feedback signals, such as dwell time or click-through activity

BCI for physiological text annotation

IUI'17Peer reviewe

The psychophysiology primer : A guide to methods and a broad review with a focus on human-computer interaction

Publisher Copyright: © 2016 B. Cowley, M. Filetti, K. Lukander.Peer reviewe

Body Matters:Exploration of the Human Body as a Resource for the Design of Technologies for Meditation

Much research on meditation has shown its significant benefits for wellbeing. In turn, there has been growing HCI interest for the design of novel interactive technologies intended to facilitate meditation in real-time. In many of these systems, physiological signals have been mapped onto creative audiovisual feedback, however, there has been limited attention to the experiential qualities of meditation and the specific role that the body may play in them. In this paper, we report on workshops with 24 experts exploring the bodily sensations that emerge during meditation. Through material speculation, participants shared their lived experience of meditation and identified key stages during which they may benefit from additional aid, often multimodal. Findings emphasize the importance of recreating mindful physical sensations during moments of mind-wandering; in particular for supporting the regulation of attention through a range of embodied metaphors and haptic feedback, tailored to key transitions in the meditation process

Extracting Relevance and Affect Information from Physiological Text Annotation

We present physiological text annotation, which refers to the practice of associating physiological responses to text content in order to infer characteristics of the user information needs and affective responses. Text annotation is a laborious task, and implicit feedback has been studied as a way to collect annotations without requiring any explicit action from the user. Previous work has explored behavioral signals, such as clicks or dwell time to automatically infer annotations, and physiological signals have mostly been explored for image or video content. We report on two experiments in which physiological text annotation is studied first to 1) indicate perceived relevance and then to 2) indicate affective responses of the users. The first experiment tackles the user’s perception of relevance of an information item, which is fundamental towards revealing the user’s information needs. The second experiment is then aimed at revealing the user’s affective responses towards a -relevant- text document. Results show that physiological user signals are associated with relevance and affect. In particular, electrodermal activity (EDA) was found to be different when users read relevant content than when they read irrelevant content and was found to be lower when reading texts with negative emotional content than when reading texts with neutral content. Together, the experiments show that physiological text annotation can provide valuable implicit inputs for personalized systems. We discuss how our findings help design personalized systems that can annotate digital content using human physiology without the need for any explicit user interaction